Staged Training Report ✓ Complete

Run ID: shoulder_session_multiheight_2
Generated: 2026-02-20 08:55:59
Stages Completed: 1
Total Elapsed Time: 10:44:40

Configuration

Config Defaults Changed Since Last Commit

ParameterPreviousCurrent
plateau_sweep.max_sweeps_per_stage 2 3
All Configuration Parameters (60 parameters)
ParameterValue
total_samples10000000
batch_size1
stage_samples_multiplier100000000000
update_interval250
window_size100
num_best_models_to_keep1
sampling_modeLoss-weighted
loss_weight_temperature0.5
loss_weight_refresh_interval50
stop_on_divergenceTrue
divergence_gap0.002
divergence_ratio1.5
divergence_patience50
divergence_min_updates10
val_spike_threshold2.0
val_spike_window15
val_spike_frequency0.75
val_plateau_patience250
val_plateau_min_delta0.0001
custom_lr0.0001
disable_lr_scalingTrue
custom_warmup-1
lr_min_ratio0.001
resume_warmup_ratio0.05
plateau_factor0.8
plateau_patience15
preserve_optimizerFalse
preserve_schedulerTrue
samples_modeTrain additional samples
num_random_obs_to_visualize2
selected_frame_offset3
runs_per_stage5
serial_runsTrue
clean_old_checkpointsTrue
enable_baselineFalse
baseline_runs_per_stage1
run_idshoulder_session_multiheight_2
enable_wandbTrue
wandb_projectdevelopmental-robot-movement
lr_sweep.lr_min1e-07
lr_sweep.lr_max0.01
lr_sweep.phase_a_num_candidates5
lr_sweep.phase_a_seeds1
lr_sweep.phase_a_time_budget_min3.0
lr_sweep.phase_a_survivor_count2
lr_sweep.phase_b_seeds3
lr_sweep.phase_b_time_budget_min10.0
lr_sweep.ranking_metricmedian_best_val
lr_sweep.min_samples_before_timeout1000
lr_sweep.min_evals_before_stop5
lr_sweep.save_sweep_stateTrue
plateau_sweep.enabledTrue
plateau_sweep.plateau_ema_alpha0.9
plateau_sweep.plateau_improvement_threshold0.0005
plateau_sweep.plateau_patience25
plateau_sweep.cooldown_updates5
plateau_sweep.max_sweeps_per_stage3
plateau_sweep.min_sweep_improvement0.0
initial_sweep_enabledTrue
stage_time_budget_min180

Timing Summary

Stage Plateau Sweeps Sweep Time Training Time Stage Total
Stage 1 12 02:48:10 00:48:51 03:37:02
TOTAL 12 02:48:10 00:48:51 03:37:02

Plateau Sweep Details

Total Sweeps: 12
Stages with Sweeps: 1 of 1
Total Sweep Time: 02:48:10
Average Sweep Duration: 00:14:00

Stage 1: 12 sweeps

LR Progression: 5.6e-04 → 1.8e-06 → 3.2e-05 → 3.2e-05 → 1.8e-06 → 1.8e-06 → 1.8e-06 → 1.8e-06 → 1.8e-06 → 1.8e-06 → 1.8e-06 → 1.8e-06 → 1.0e-07

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 16,000 00:05:11 1.78e-06 00:14:01
2 26,750 00:22:44 3.16e-05 00:14:03
3 39,500 00:40:58 3.16e-05 00:14:01
4 53,000 00:59:24 1.78e-06 00:13:57
5 62,750 01:16:34 1.78e-06 00:13:59
6 71,500 01:33:26 1.78e-06 00:14:02
7 82,750 01:51:08 1.78e-06 00:14:03
8 95,750 02:09:25 1.78e-06 00:13:59
9 107,250 02:27:09 1.78e-06 00:14:02
10 121,250 02:45:47 1.78e-06 00:13:58
11 135,250 03:04:20 1.78e-06 00:13:58
12 141,750 03:20:26 1.00e-07 00:14:01

Stage Results

Stage Best Loss Stop Reason Samples Trained Time Sweeps LR (Initial→Final)
Stage 1 0.033306 max_sweeps (3) 7,750 03:37:02 12 5.6e-04→1.0e-07

Total Plateau Sweeps: 12

Stop Reason Breakdown

Loss Across Full Training Run

Loss Detail (Post Initial Drop)

Multi-Run Statistics

Total Runs: 5
Average Best Loss: 0.049978 ± 0.011243
Best Overall: 0.033306
Worst Overall: 0.061324

Stage 1 (5 runs)

Run Best Loss Stop Reason Samples Time Selected
1 0.033306 max_sweeps (3) 7,750 03:37:02
2 0.055669 max_sweeps (3) 9,250 01:27:51
3 0.061324 max_sweeps (3) 14,750 01:33:18
4 0.059616 max_sweeps (3) 6,500 01:55:38
5 0.039976 max_sweeps (3) 12,000 01:54:27
Mean: 0.049978 ± 0.011243 Min: 0.033306 / Max: 0.061324 Range: 0.028018

Best Checkpoint

Name: best_model_auto_session_so101_multiheight_part1_1345_shoulder_session_multiheight_2_00143250_cont_val_0.033306.pth
Stage: 1
Hybrid Loss (full session): 0.054795

Learning Rate Timeline with Plateau Sweeps

Stage Progression

Stage Orig Loss Train Loss Time Samples Stop Reason
1 ⭐ 0.054795 0.033306 03:37:02 7750 max_sweeps (3)

Hybrid Loss Over Original Session (per Stage)

Stage 1 (Best) - Hybrid Loss: 0.054795

Sample Counts

Cumulative Across All Stages

Per Stage

Stage 1 (Best) - Total Samples: 7,750

Best Checkpoint Inference

Selected Frame 3

Action 0

Action 1

Action 2

Random Observations

Observation 629

Action 0
Action 1
Action 2

Observation 579

Action 0
Action 1
Action 2